REMARKS 

Claims 1, 3-23, 25-40 and 42-47 are in this application and are presented for 
consideration. By this Amendment, Applicant has amended claims 1, 3-23, 25-40 and 42-44. 
Applicant has also added new dependent claims 45 and 46 and independent claim 47. Claim 
41 has been canceled. 

Claims 1, 3-23 and 25-44 have been rejected under 35 U.S.C. 112, second paragraph, 
as being indefinite for failing to particularly point out and distinctly claim the subject matter 
which Applicant regards as the invention. 

Applicant has amended the claims to adapt them to the U. S. style. It is Applicant's 
position that the claims as now presented arc clear and satisfy the requirements of the statute. 

Claims 1, 3-23 and 25-44 have been rejected under 35 U.S.C. 1 12, first paragraph, as 
failing to comply with the enablement requirement. The Office Action takes the position that 
Applicant's specification does not support the claim limitation of a determination of a position 
and an orientation or pose of the image receiving unit as found in claim 1 . Further, the Office 
Action states that Applicant's specification does not the claim limitation of claim 1 that at least 
one robot-specific coordinate system being faded into the image of the real environment or the 
claim limitation of claim 23 of a display of at least one robot-specific coordinate system. 

Applicant has amended claim 1 to provide that the step of obtaining a position and an 
orientation or pose of the image receiving unit. This is clearly supported on page 17, first 
paragraph of the specification. 

Applicant respectfully traverses the rejection regarding claims 1 and 23. The 
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specification clearly supports that the robot-specific coordinate system is superimposed into the 
image of the real environment. The specification states the following on page 24, lines 4- 1 3 of 
the first full paragraph: 

Figs. 7a to 7c show possible representations on the viewing device 1.1 with areas 
of the real environment, such as in particular a robot 7 or a workpiece 8 being 
machined by it, as well as robot-specific, computer-generated information faded 
into this image of the real environment and connected thereto. It is possible with 
the image of a real robot 7 with a tool 7.1, such as welding tongs, to generate 
coordinate systems to be associated therewith, such as the stationary world 
coordinate system Kl with the coordinates X+, Y+, Z+. 
Further the specification states the following on page 25, lines 7-9: 

Fig. 7c shows the fading in of a tool coordinate system K3 with the coordinates 
X3+, Y3+, Z3+ as AR assistance in manual cartesian movement. 
The Figures 7a-7c clearly show the viewing device with robot-specific coordinate systems faded 
into the image of the real environment. As such, it is Applicant's position that claims 1 and 23 
contain subject matter which is sufficiently described in the specification in such a way as to 
enable one skilled in the art to use the invention. Applicant respectfully requests that the 
Examiner favorably consider claims 1 and 23 as now presented. 

Claims 1, 3-23 and 25-44 have been rejected under 35 U.S.C. 1 12, second paragraph, 
as being indefinite for failing to particularly point out and distinctly claim the subject matter 
which Applicant regards as the invention. 
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Applicant has amended the claims to remove the term "faded" and replace it with the 
term "superimposed". It is Applicant's position that the claims as now presented are clear and 
satisfy the requirements of the statute. 

Claims 23, 25-37, 39 and 41-44 have been rejected under 35 U.S.C. 102(e) as being 
anticipated by Watanabe et al. (US 6,763,284). 

The present invention relates to a device for visualizing computer-generated information 
in an image of the real environment. The present invention processes robot-specific, computer- 
generated information for the purpose of displaying the information on a viewing device. An 
image receiving means detects an image of a robot and the real environment surrounding the 
robot. The images are displayed on a viewing device. The robot-specific information is 
superimposed on the robot and the real environment images so that the real image and the 
information to be displayed are jointly displayed on the viewing device. A control unit controls 
the robot based on the robot-specific information such that the robot moves based on 
movement of the image of the robot via the control means. This advantageously allows a user 
to move the virtual image of the robot such that the actual robot moves in response to the 
movements of the virtual image. 

The present invention solves the problem of making it easier for an operator of a robot 
to program the robot or to move the robot in a desired direction. Often times it is not clear to 
a user which reference coordinate system has been selected for the robot or how the coordinate 
system is oriented in space. This can lead to serious damage or even injuries if an inexperienced 
user is operating the robot. The present invention advantageously solves this problem by 
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making apparent the motion of the robot to the operator. This is achieved by optically 
representing information to the operator. This information is not perceptible or is insufficiently 
perceptible and cannot be seen. With the invention, the robot image and robot specific 
information are presented in a combined format. Additional information that is decisive for 
moving the robot is provided to the operator via this optical representation so that the operator 
can better understand the behavior of the robot. This advantageously allows the user to move 
the robot more easily and in a more purposeful manner so that unnecessary movements are 
avoided. The prior art as a whole fails to disclose such features or advantages. 

Watanabe et al. discloses a robot teaching apparatus. Mounted at an arm distal end of 
a robot 5 are a camera 2 and a projector 1 . The projector 1 projects a pattern of light. A robot 
controller 6 houses an image processing device therein and it is connected to the robot 5, the 
projector 1, the image pickup device 2, a display device 3 and a teaching operation panel (not 
shown). The robot 5 is moved with a manual operation to a position and an attitude at which 
it is properly spaced from a reference work 7 and it looks over the entire working space. An 
image including the reference work 7 in a field of view is picked up by the camera and the 
image is displayed on the image display device 3. A 2-dimensional image position pointing 
device 4 utilizing a mouse is provided in the image display device 3 and an operator points to 
a position (uO, vO) to be measured by the robot 5 on the image using the pointing device 4. The 
pointed position (uO, vO) is stored in the robot controller 6. The pointing device 4 is moved 
along a plurality of points to constitute a working line on an image. After the measurement 
starting point is on the working line, a view line corresponding to the position (uO, vO) is 
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obtained using the position (uO, vO) on the 2-dimensional image and the position and orientation 
of the image pickup device 2. A slit light is projected from the projector 1 onto the reference 
work 7 and measurement of a 3-dimensional position on the reference work 7 is started. An 
image, including an image of the bright line formed on the reference work 7, is photographed 
by the camera 2. The robot is moved along a working path a small distance and light projecting 
and photographing are performed for each small distance so that 3-dimensional position points 
at which the bright line crosses the working line are sequentially obtained. The detection results 
are verified by superimposing the working line on to the image of the camera 2 on the screen 
of the image display device 3. 

Watanabe et al. fails to teach or suggest the combination of a control means that 
controls a robot based on robot- specific information superimposed on an image of the robot on 
a viewing device. In the present invention, the movement of the robot corresponds to the 
movement of the robot image applied by the user. At most, Watanabe et al. merely discloses 
that an image including a bright line image on a reference object is photographed and 3- 
dimensional positions of points are sequentially measured along a working line so that a 
movement path of a robot is created using the 3-dimensional positions. The present invention 
takes a different approach. The present invention provides a viewing device for superimposing 
abstract information that is not visible to the operator on an image of the robot so that the 
operator has a visual representation of the robot-specific conditions actually occurring. The 
control means of the present invention advantageously allows the operator to move the 
displayed image of the robot so that the actual robot corresponds to the movements of the 



18 



virtual image. This advantageously allows the movement of the robot to be facilitated to the 
operator so that the operator understands in which direction the robot is moving. In contrast 
to the present invention, Watanabe et al. fails to disclose that an actual image of the robot 5 is 
displayed on the image display device 3 as featured in the claimed combination. This is a 
significant difference because the image of the robot on the viewing device is necessary so that 
the information necessary for moving the robot, such as coordinate axes, can be superimposed 
on the image of the robot. This advantageously allows the operator to more easily understand 
how to move the robot in a desired direction. Watanabe et al. does not provide such an 
advantage since image display device 3 fails to display an image of robot 5 so that an operator 
can move the robot image so that the movement of the robot corresponds to the movement of 
the virtual image of the robot applied by the operator. As such, the prior art as a whole teaches 
a different approach and fails to suggest the features or advantages of the present invention. 
Accordingly, Applicant respectfully requests that the Examiner favorably consider claim 23 as 
now amended and all claims that depend thereon. 

Claims 1,3-15, 17 and 19-22 have been rejected under 35 U.S. C. 103(a) as being 
unpatentable over Watanabe et al. in view of Huissoon (US 6,044,308) and Roos (US 
6,615,112). 

Huissoon discloses a method and device for robot tool frame calibration. A robot 30 
comprises a base 32 and a manipulator 34. The manipulator 34 is provided with several 
linkages 36 connected to joints 38 with an end effector 40. A tool having a tool center point 
(TCP) is attached to end-point 40 of robot 30 and is robotically controlled by the robot. The 
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pose of coordinate frame E of the robot end-point 40 with respect to a global coordinate 
reference frame G of robot 30 is defined by a forward kinematic model of the robot so that 
given the angels of joints 38 and the lengths of links 36 and the robot geometry, the pose of 
frame E can be computed with respect to frame G. A reference fixture 44 is mounted on a 
calibration jig 60 comprising a support table 62 and one or more active area sensors 64 are 
attached to the support in known positions with respect to the reference fixture 44. The sensors 
64 are located below the viewing window 66 located in table 62. A laser grid generator 63 
illuminates the edges of a gas cup and the tip of a contact tube so that the analysis area sensor 
64 image data can establish the location of TCP 42 with respect to reference fixture 44. The 
TCP is the focal point of the welding laser and its position is determined by analyzing the image 
of the focusing laser spot on a projection screen 66. Since the TCP sensor is in a known 
position with respect to the structured light sensor calibration features, the TCP position with 
respect to the sensor can be computed. 

Roos discloses a method and device for multistage calibration of multiple-axis 
measuring robots 6 and associated optical measuring devices 10 in a measuring station 1 for a 
workpiece 2. Calibration occurs in a measuring cascade comprising three calibrating steps. The 
optical measuring device 10 and the operating point 28 thereof, the manipulator 6 and the axes 
thereof and the allocation of the manipulator 6 with respect to the workpiece 2 are successively 
calibrated. 

As previously discussed above, Watanabe et al. fails to teach or suggest the combination 
of controlling a robot based on robot-specific information superimposed over a virtual image 
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of the robot such that the robot moves based on movement of the virtual robot image. At most, 
Watanabe et al. discloses that a path is superimposed onto an image of a workpiece and that 
a user is asked to set the points of the path manually to verify detection results. The present 
invention takes a different approach. The present invention provides a viewing device for 
superimposing abstract information that is not visible to the operator on an image of the robot 
so that the operator has a visual representation of the robot-specific conditions actually 
occurring. Watanabe et al. fails to disclose that an actual image of the robot 5 is displayed on 
the image display device 3 so that the movement of the image of the robot correspond to the 
movement of the actual robot as featured in the claimed combination. 

Further, Roos and Huissoon fail to provide any suggestion for moving an actual robot 
based on movement of a virtual image of the robot. Huissoon merely discloses a kinematic 
model to determine the pose of frame E with respect to frame G. Huissoon does not disclose 
that the movement is based on the actual representation of a robot as detected by an image 
receiving unit as claimed. Huissoon fails all together to disclose a viewing device that displays 
an image of a robot that is detected from an image receiving device as claimed. Roos discloses 
successively calibrating the optical measuring device 1 0 and the operating point 28 thereof, the 
manipulator 6 and the axes thereof and the allocation of the manipulator 6 with respect to the 
workpiece 2. Roos fail to provide any suggestion of moving an actual robot based on a virtual 
image of the robot. In fact, Roos and Huissoon fail to suggest providing any type of real robot 
image or real environment image. As such, the prior art as a whole takes a different approach 
and fails to suggest the features of the claimed combination. Accordingly, Applicant 
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respectfully requests that the Examiner favorably consider claim 1 and all claims that depend 
thereon. 

Claim 16 has been rejected under 35 U.S.C. 103(a) as being unpatentable over 
Watanabe et al. in view of Huissoon and Roos. As previously discussed above, the references 
as a whole fail to suggest the combination of features claimed. Specifically, the references fail 
to teach the combination of controlling a robot based on robot-specific information 
superimposed over a virtual image of the robot such that the robot moves based on movement 
of the virtual robot image. The references together do not suggest the combination of features 
claimed. One of ordinary skill in the art is presented with various concepts, but these concepts 
do not provide any direction as to combining the features claimed. All claims define over the 
prior art as a whole. 

Claim 18 has been rejected under 35 U.S.C. 103(a) as being unpatentable over 
Watanabe et al. in view of Mizuno et al. (US 5,876,325), Huissoon and Roos. 

Although Mizuno et al. teaches a surgical manipulation system, the references as a 
whole fail to suggest the combination of features claimed. Specifically, the references provide 
no teaching for the combination of controlling a robot based on robot-specific information 
superimposed over a virtual image of the robot such that the robot moves based on movement 
of the virtual robot image. The references together do not suggest the combination of features 
claimed. One of ordinary skill in the art is presented with various concepts, but these concepts 
do not provide any direction as to combining the features claimed. All claims define over the 
prior art as a whole. 
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Claim 38 has been rejected under 35 U.S.C. 103(a) as being unpatentable over 
Watanabe et al. As previously discussed above, Watanabe et al. does not teach moving a robot 
moves based on movement of the virtual robot image. As such, the reference provides a 
different approach and does not suggest the features of the claimed combination. Accordingly, 
all claims define over the prior art as a whole. 

Claim 40 has been rejected under 35 U.S.C. 103(a) as being unpatentable over 
Watanabe et al. in view of Mizuno et al. Although Mizuno et al. teaches a surgical 
manipulation system, the references as a whole fail to suggest the combination of features 
claimed. Specifically, the references provide no teaching for the combination of controlling a 
robot based on robot-specific information superimposed over a virtual image of the robot such 
that the robot moves based on movement of the virtual robot image. The references together 
do not suggest the combination of features claimed. One of ordinary skill in the art is presented 
with various concepts, but these concepts do not provide any direction as to combining the 
features claimed. All claims define over the prior art as a whole. 

Claims 1 and 23 have been rejected under 35 U.S.C. 102(e) as being anticipated by 
Saito et al. (US 6,587,752). 

Saito et al. discloses a robot operation teaching method and apparatus including a three- 
dimensional measuring system that measures spatial coordinates corresponding to points 
designated on camera images. A display is provided to show a space image from camera or 
cameras overlaid by an image of a geometric model corresponding to the space image. A 
pointing device having at least two degrees of freedom is used to define work trajectories by 
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preparing, in a model space, simple geometric elements corresponding to the actual space 
image. By using parametric modeling to supply the definitions of geometric elements in 
advance, the geometric elements can be adapted for other tasks by modifying some of the 
geometric element assignments and parameters associated with the definitions. 

Saito fails to teach and fails to suggest the combination of a control means that controls 
a robot based on robot-specific information superimposed on an image of the robot on a 
viewing device. In the present invention, the movement of the robot corresponds to the 
movement of the robot image applied by the user. At most, Saito discloses displaying a model 
space image on a corresponding actual image obtained from a camera. However, Saito does 
not disclose that an image of a robot is captured and robot-specific information is superimposed 
on the virtual image of the robot so that an operator can move the virtual image such that the 
operator's movement of the virtual robot image moves the actual robot. Saito merely teaches 
a space image from a camera that is overlaid by an image of a geometric model. In contrast to 
Saito, the present invention takes a different approach. The present invention provides a 
viewing device for superimposing abstract information that is not visible to the operator on an 
image of the robot so that the operator has a visual representation of the robot-specific 
conditions actually occurring. The control means of the present invention advantageously 
allows the operator to move the displayed image of the robot so that the actual robot 
corresponds to the movements of the virtual image. This advantageously allows the operator 
to view the movement of the robot so that the operator understands in which direction the robot 
is moving. Saito fails to suggest a control means as featured in the claimed combination. As 
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such, the prior art as a whole teaches a different approach and does not suggest the features as 
claimed. Accordingly, Applicant respectfully requests that the Examiner favorably consider 
claims 1 and 23 as now presented. 

Applicant has added new dependent claims 45 and 46, which are based on claims 1 and 
23 , respectively. New dependent claims 45 and 46 further clarify the robot-specific information. 
New independent claim 47 provides for similar features as claim 1 , but further clarifies the 
robot-specific information. Applicant respectfully requests that the Examiner favorably 
consider new claims 45 and 46. 

Favorable action on the merits is requested. 



Respectfully submitted 
for Applicant, 
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